Does usage of monetary incentive impact the involvement in surveys? A systematic review and meta-analysis of 46 randomized controlled trials

Background Surveys are an effective method for collecting a large quantity of data. However, incomplete responses to these surveys can affect the validity of the studies and introduce bias. Recent studies have suggested that monetary incentives may increase survey response rates. We intended to perform a systematic review and meta-analysis of randomized controlled trials (RCTs) to evaluate the effect of monetary incentives on survey participation. Methods A systematic search of electronic databases was conducted to collect studies assessing the impact of monetary incentives on survey participation. The primary outcome of interest was the response rates to incentives: money, lottery, and voucher. We used the Cochrane Collaboration tool to assess the risk of bias in randomized trials. We calculated the rate ratio (RR) with its 95% confidence interval (95% CI) using Review Manager Software (version 5.3). We used random-effects analysis and considered the data statistically significant with a P-value <0.05. Results Forty-six RCTs were included. A total of 109,648 participants from 14 countries were involved. The mean age of participants ranged from 15 to more than 60 years, with 27.5% being males, 16.7% being females, and the other 55.8% not reported. Our analysis showed a significant increase in response rate in the incentive group compared to the control group, irrespective of the incentive methods. Money was the most efficient way to increase the response rate (RR: 1.25; 95% CI: 1.16,1.35; P = < 0.00001) compared to voucher (RR: 1.19; 95% CI: 1.08,1.31; P = < 0.0005) and lottery (RR: 1.12; 95% CI: 1.03,1.22; P = < 0.009). Conclusion Monetary incentives encourage the response rate in surveys. Money was more effective than vouchers or lotteries. Therefore, researchers may include money as an incentive to improve the response rate while conducting surveys.


Introduction
Surveys allow researchers to collect a large quantity of data efficiently. It can be widely applied to collect information like participants' demographics, knowledge, past behaviors, and opinions. Surveys are a reliable and precise method for data gathering as they provide all the participants with standardized and uniform questions. Surveys can be conducted in different ways, such as written questionnaires, face-to-face or telephone interviews, and online (email or website) surveys.
The generalizability of survey results depends mainly on response rate; refusing to respond to surveys could lead to nonresponse bias which occurs when there is a significant difference between participants who responded to the survey and those who did not. It has a negative effect on the reliability and validity of survey study findings [1]. Achieving adequate response rates is a significant obstacle in survey research. Therefore, efforts are continuously made to improve survey response rates using different methods like incentives. Various incentives have been used to increase response rates, such as candy, lottery, vouchers, and money [2][3][4].
Non-monetary incentives like shopping vouchers are commonly used to improve survey response, but it has little to no impact on the survey rate. On the other hand, monetary stimulus has successfully increased response rates. For example, one systematic review found the survey response rate doubled upon receiving monetary incentives [5], and another showed that using incentives increased participation in clinical research [6]. We conducted a systematic review and meta-analysis of randomized controlled trials (RCTs) to evaluate the overall effectiveness of incentives like the lottery, vouchers, and money in enhancing response rates to surveys.

Methods
We followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement and Cochrane Handbook for Systematic Reviews of Interventions [7,8] S1 File.

Data sources and search strategy
We performed a comprehensive and systematic search of the PubMed, Web of Science, Scopus, Embase, and Cochrane library databases from inception to the 23 rd of September 2021. A manual updated search was conducted at the end of November 2021. There were no language or publishing date restrictions. A combination of keywords and standardized index terms was used to generate the search strategies. We used keywords like "payments", "incentive", "response", "participation", "enrollment", and "randomized" and their synonyms in our search. The full research strategy and results in different databases are reported in S1 Table.

Study selection and eligibility criteria
The criteria of study selection we used to select studies in this meta-analysis were as follows: (1) study design: randomized controlled trials;(2) the study investigated the effects of incentive compared to no incentive on participation in surveys;(3) there were no specific criteria to the study participants;(4) the outcome of interest was the response rate. We excluded nonrandomized trials and articles without relevant population, intervention, or outcomes.
Two reviewers (AH and MAR) independently screened and selected studies using the eligibility criteria. Any disagreements were resolved by discussion with third reviewers (KSA and NAE).

Data extraction
Data were extracted using a formatted excel sheet, including the first author's last name, year of publication, country, the number of reminders, population criteria, incentive types, sample size, age, and gender. All data were separately extracted by (AH and MAR) and integrated clearly. Disagreements were solved through discussion between reviewers (KSA and NAE).

Risk of bias assessment
To assess the bias in all included RCTs, two reviewers (AH and MAR) used the Cochrane Collaboration's tool for assessing the risk of bias in randomized trials [9], which covers biases including selection bias, performance bias, attrition bias, detection bias, reporting bias, and other biases. Each domain's risk of bias was categorized as high, unclear, or low risk of bias. Any conflicts were resolved by consulting a third reviewer (KSA and BA).

Outcomes of interest
The primary outcome of interest was the response rates according to the types of incentives: money, lottery, and voucher. Data about response rate was extracted in the format of event/ total for both incentive and control groups.

Statistical analysis
We used Review Manager Software (version 5.3) to perform the meta-analysis [10]. We calculated the risk ratio (RR) with its 95% confidence interval (95% CI) using event and total numbers. We considered the data statistically significant if the P-value was less than 0.05. We used I 2 a to evaluate the heterogeneity between the included studies by random-effects analysis. Heterogeneous was considered to be low if I 2 < 50%. We performed our analysis based on the type of incentive; money, lottery, or voucher. We assessed publication bias throughout the included studies for money used in response rate [11]. A funnel plot was used by plotting the risk ratio (RR) on the x-axis and the log of risk ratio on the y-axis.

Study identification and selection
The search retrieved 11,693 articles. After 5,412 duplicates were removed, 6,281 articles were screened. We excluded 5,760 at the title/abstract screening stage as they were not eligible. The remaining 521 articles underwent a full-text evaluation to determine eligibility. Finally, 46 RCTs [3,4, met the criteria for final inclusion in our systematic review and meta-analysis. The process of study selection and the reasons for exclusion are shown in Fig 1.

Characteristics of included studies
Our search identified 46 studies with 109,648 participants from 14 countries; 27.5% were males, 16.7% were females, and 55.8% were not reported. The average age ranged from 15 to more than 60 years. The duration of the questionnaire ranged from one week to one year. The

PLOS ONE
summary of the included studies and baseline characteristics of participants are shown in Table 1.

Risk of bias of the included studies
All of the included RCTs demonstrated a low risk of bias in performance and detection biases. Attrition bias was a high risk of bias in three RCTs [33,35,41], unclear risk of bias in Boucher et al. [14], and low risk of bias in the rest of the included RCTs. Selective reporting was low risk in all included RCTs, except three RCTs were at high risk of bias [4,28,35]. The risk of bias summary for each study and the risk of bias graph for bias domains is shown in Fig 2

Outcome of interest
Our analysis showed a significant increase in response rate in the incentive group compared to the control group, irrespective of the incentive methods. Money was associated with the highest increase in response rate (RR: 1.25; 95% CI: 1.16,1.35; P = < 0.00001) (Fig 3) compared to voucher (RR: 1.19; 95% CI: 1.08,1.31; P = < 0.0005) ( Fig 4A) and lottery (RR: 1.12; 95% CI: 1.03,1.22; P = < 0.009) ( Fig 4B). The funnel plot for the publication bias had the traditional conal appearance with good bilateral symmetry. Outliers were removed from the analysis and did not affect the pooled RR, suggesting minimal publication bias ( Fig 5).

Discussion
We included 46 RCTs to evaluate the effect of using monetary incentives to increase the participants' response rate to the survey. Our results showed that using the incentive increased the response rate to surveys. Money was the most effective way, followed by vouchers and lottery.
Monetary incentives have been widely used to enhance response rates to questionnaires, mainly postal surveys. A large systematic review using mailed questionnaires reported that the odds of response to the questionnaire were doubled by using a monetary incentive [56]. Other factors that improve post-mail surveys response include short questionnaires, colored ink, personalized letters, certified mail with a return receipt, and follow-up contact [56]. However, the available evidence on using monetary incentives in electronic/online surveys is limited [57]. Few RCT studies reported that monetary incentives boost the response rate in online surveys compared to no incentives [21,45,54]. Surprisingly, a study found that other factors could be as effective as a monetary incentive in increasing the response rate to online surveys, such as well-written short questions, easy accessibility, and engaging topics [19].
Online surveys have a lower response rate compared to postal surveys [58,59] for several reasons; enrollment of the study sample is challenging compared to postal surveys due to the unavailability of email addresses for all potential invitees and difficulty reaching certain types of participants such as those who do not have internet access [60] also trust is a large obstacle for internet survey participation as invitees may be hesitant to respond because of fears of potential scams, or malicious links infected with computer viruses [61]. However, online surveys have potential advantages by reducing printing and postage costs, branching questions, and adhering to a particular question format (e.g., selecting only one) [62]. Although internet surveys are less effective than other survey methods, combining multiple-mode surveys (e.g., Internet and mail) may improve overall survey response rates [63].
Prepaid incentives (unconditional incentives) are more effective in increasing response rates in comparison to payment after survey completion (conditional incentives) [64,65]. This could be explained by social exchange theory as providing participants monetary incentives in advance encourages them because they feel they should reciprocate for the reward they receive by completing the survey. On the other hand, promising a gift or money to invitees only after they answer survey questions does not give them the urge to accept the survey invitation [62].
A meta-analysis of 238 experiments concluded that prepaid cash rewards significantly increased the survey response and completion rate compared to contingent monetary incentives upon completion or return of the survey [66]. The prepaid incentive might be challenging to achieve in a web survey, as it is difficult to couple the incentive with the survey request, as reported by [67]. Young et al. confirmed that unconditional incentives had the most significant effect, but they reported that the conditional approach was more cost-effective [54]. A more recent RCT by Cheung et al. found that combining prepaid and promised incentives (mixed incentives) was superior to the promised incentive by increasing the retention rate by 48% [16].

PLOS ONE
The correlation between incentive value and response rate remains unclear. A systematic review of trials suggests a non-linear relationship between the size of the incentive and the improvement in response [68]. All incentives resulted in a higher response rate than no incentive; the US$2 amount was the most cost-effective. However, the US$10 incentive achieved the highest response rate. Also, cash was more effective than other incentives like prize draws or lotteries [16].
Incentives may enhance response rates and reduce the likelihood of nonresponse error [23]; nevertheless, few studies claim that higher response rates do not necessarily indicate a reduction in nonresponse bias [62,69]. In addition to that, concerns that incentives may actually introduce response bias by being more appealing to those with lower socioeconomic status [38,45]. Also, previous research found that women are more likely than men to participate in surveys, so using monetary incentives may exacerbate the overrepresentation of women, raising the risk of response bias. Further studies are required to test the effect of incentives on response and nonresponse bias, plus analyze the impact of different reward sizes on response rates.
Incentives may affect the survey response quality. For example, motivating members to respond who otherwise would have refused will decline response quality and jeopardize the survey outcomes. On the other hand, offering incentives will lead to better quality answers and hence improve survey outcomes. Based on the currently available studies, offering incentives are unlikely to change or alter the quality of the survey [70].
There are a few limitations to our study. Most of the included studies were performed in developed countries like the United States, the United Kingdom, and European countries. The applicability of the results in developing countries needs to be confirmed by enrolling more studies. Allocation concealment bias was unclear in most of the included RCTs, which can introduce bias into our results. We only investigated the effect of incentives on the response rate, but we did not examine the accuracy and reliability of the data collected. More studies are needed to evaluate the effect of incentives on the quality of response. We did not evaluate the relationship between the amount of the incentive offered and the response rate or if there is any minimum amount of monetary incentive that should be other to the participant.

Conclusions
In conclusion, using monetary incentives is associated with increasing the response rate while conducting a survey. Using money was associated with a higher response rate than vouchers or lottery. Therefore, researchers should include money as an incentive to increase the response rate to the survey.